Click  here  to  view  PowerPoint  presentation;  Press  Esc  to  exit 


ORGANIZATION 


Visualizing  Non-Physical,  Logical  Constructs 
for  Command  Decision  Support 


James  Llinas,  Ph.D 

315  Bell  Hall 
University  at  Buffalo 
Buffalo,  NY  14260 
USA 

llinas@eng.buffalo.edu 


T.  Kesavadas,  Ph.D 

315  Bell  Hall 
University  at  Buffalo 
Buffalo,  NY  14260 
USA 

kesh@eng.buffalo.edu 


MOTIVATION 

The  motivation  for  this  Provocation  Topic  presentation  is  directly  related  to  existing  critical  issues  and 
challenges  in  both  the  Data  Fusion  and  Visualization  communities.  Both  of  these  communities  are 
wrestling  with  the  meanings  (ontologies)  of,  and  means  for  visualization  of  what  we  are  calling  “logical 
constructs”,  or  constructs  that  have  relevant  semantic  meaning  in  an  application  context  but  do  not 
correspond  to  an  observable,  measurable  reality  in  the  real  world.  Such  constructs  correspond  to  what  the 
Data  Fusion  community  calls  “Level  2”  and  “Level  3”  processing,  processes  that  produce  results  which 
are  called  “situations”,  “threats”,  “intentions”,  “operational  readiness”  and  the  like.  Limited  research  in  the 
Data  Fusion  community  at  these  Levels  has  resulted  in  a  generally  poor  and  unstructured  understanding  of 
what  these  constructs  really  are  and  how  they  might  be  ontologically  structured  and  related  to  one  another. 
Even  if  such  definitions  and  categorizations  were  known,  there  is  the  subsequent  question  of  how  to 
communicate  these  mental  constructs  to  a  human  such  that  they  can  be  “seen  in  the  mind”,  as  described  in 
the  Call  for  Participation. 

The  reason  these  issues  are  important  is  that  these  “logical  constructs”  are  the  informational  states  around 
which  higher-level  command  decision-making  occurs;  e.g.,  force-level  maneuver  decisions  will  depend  on 
fused  estimates  of  a  hostile  “threat”  state,  whereas  lower-echelon  decisions  such  as  to  shoot  at  a  specific 
target  depend  instead  on  fused  estimates  of  a  physical  target’s  location  in  space,  corresponding  to  a 
physical  reality.  Without  formalized  definitions  of  the  set  of  logical  constructs,  Data  Fusion  and 
Visualization  systems  will  be  developed  in  inconsistent  ways  and  will  have  irregular  payoff  and  benefit 
to  the  upper  command  levels  of  the  operational  military.  We  propose  that  the  Workshop  address  the 
various  and  complex  issues  dealing  with  the  topic  of  “logical  constructs”,  and  the  means  by  which  the 
NATO  community  can  develop  methodologies  and  architectures  for  visualizing  these  “non-physical, 
non-geospatial/temporal”  constructs. 


OBJECTIVES 

In  recent  years  research  in  Data  Fusion  and  Visualization  science  has  focused  on  understanding  physical 
environments  and  data  types.  Advanced  visualization  techniques  including  VR  and  other  related 
technologies  have  succeeded  in  providing  meaningful  outputs.  However  many  abstract  concepts  are 
beyond  traditional  modes  of  display  and  hence  require  new  paradigms  in  visualization. 

Our  objective  is  to  present  an  organized,  provocative  introductory  presentation  regarding  the  definitions 
and  ontological  structure  of  such  non-physical  concepts  as  well  as  possible  means  for  visualizing/ 
communicating  their  states  as  a  framework  to  encourage  lively  discussions  among  military  staff  and 
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ORGANIZATION 


research  scientists  on  approaches  for  both  computing  the  estimated  value  of  and  for  efficiently 
communicating/visualizing  the  results  of  fusion-based  estimates  of  these  conceptual  states.  Through  such 
discussion,  it  is  hoped  that  a  NATO  consensus  can  be  established  on  the  issues  surrounding  this  important 
topic,  and  also  that  agreement  can  be  established  on  additional  research  needed  to  both  better  understand 
this  topical  area  and  to  develop  effective  visualization/communication  methods. 


TECHNICAL  APPROACH 

We  are  proposing  to  come  to  the  Workshop  with  a  presentation  that  elaborates  on  these  issues.  The  first 
point  is  to  define  further  what  is  meant  by  “non-physical  entities”;  in  this  case  we  plan  to  elaborate  on  the 
example  of  “Operational  Readiness”,  and  to  discuss  what  is  the  modem  interpretation  of  this  term  and  the 
associated  concepts.  This  will  lead  into  a  sampling  of  a  draft  ontological  construct  for  this  term,  showing 
that,  as  for  many  such  constructs,  it  is  composed  of  both  entities  that  do  have  a  physical  reality  but  also 
notional  terms  that  result  from  abstract  and  fuzzy  constructs.  Given  this,  the  challenge  of  visualization  is 
elaborated  on,  to  show  that  there  are  at  least  two  major  challenges  to  visualization:  the  dimensionality  of 
these  constructs,  and  the  non-geospatial,  non-temporal  elements  of  them.  It  is  also  argued  that  the  need  for 
an  ontological-level,  formally-constructed  characterization  of  these  constructs  is  central  to  a  consistent 
systems-level  approach  to  the  design  of  the  overall  information  fusion  process.  Further,  it  is  shown  by 
example  that  such  ontologically-based  design  methodologies  are  not  precedent-setting  for  defense-type 
applications.  Another  factor  discussed  is  the  distinction  between  the  user’s  mental  model  and  the 
visualization-construct,  i.e.  the  “display”.  Mental  models  are  usually  defined  as  “deep”  constructs, 
reflecting  the  comprehensive  understanding  that  a  human  has  about  a  given  process  or  object;  the 
computed  and  visualized  product  can  be  thought  of  as  an  instance  of  that  model,  but  as  a  result  the 
delivered  visualization  should  draw  on  the  mental  model  that  a  user  has.  But  even  if  that  visualization  is 
consistent  in  this  way,  there  is  still  the  question  of  how  the  user  visualizes  his  mental  model  in  his  mind  - 
one  challenge  or  hypothesis  to  explore  is  whether  the  “optimal”  visualization  is  a  construct  that  mirrors  the 
user’s  mental  “image”  of  his  model,  or  whether  these  two  entities  are  separate  and  reside  in  their  own 
separate  contexts  (i.e.  computer-screen  and  human  mind).  A  nagging  question  is  also:  what  is  “the” 
authoritative  taxonomical  structure  from  which  we  should  build  the  relevant  ontology?  By  this  is  meant 
that,  if  the  US  defense  community  is  typical,  there  are  many  lists  of  vocabulary  and  terminologies  that 
abound  in  the  defense  community  -  if  it  is  agreed  that  in  fact  an  ontological  framework  for  the  terms  of 
interests  is  needed,  which  list  is  the  starting  point?  In  this  regard,  some  examples  of  the  US  community’s 
“Essential  Elements  of  Information”  or  “EEI’s”  are  described  along  with  some  limited  ontological- 
structuring  of  these  terms  that  has  been  carried  out.  Finally,  we  give  two  major  examples  of  visualization 
techniques  that  have  been  used  in  defense-type  applications  as  exemplars  of  some  modern-day 
display  constructs:  these  are  the  “Event  Wall”  and  the  “Starlight”  systems.  These  systems  have  been 
designed  with  the  idea  of  showing  information  of  high-dimension  and  information  having  complex 
interrelationships. 
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Visualization,  Info.  Fusion  and  Logical 

Constructs 


INTEL 


SATELLITE 


RAW 

DATA 


FOCUS 


Level  1,  2  3 . Fusion  -  Estimates 


Transforms  -  Modalities  Mental  Models  Ontology 


Physical 
Traditional  Viz. 


Non  physical 


Non  Traditional 


2-D  Screens 


3-D  stereo  glasses  HMD  CAVE 


Audio  Haptics 


Directions  in  “Level  2,3” 
Information  Fusion 


7-3 


Improved  Standardization 
And  Formalism  in 
Level  2,3  Info  Fusion 
Engineering  Methods 


Motivation 


Higher-echelon  command  decision-making  depends  on 
information  at  the  conceptual  level,  involving  such  terms  as: 

-  “Center  of  Gravity”,  “Situation”,  Intent”,  “Course  of 
Action”,  “Operational  Readiness” . etc 

An  integrated  and  adequately-formalized  consensus  set  of 
definitions  for  these  terms,  an  Ontology ,  suitable  for 
automation,  does  not  exist. 

Such  states  are  ideally  estimated  via  Information  Fusion 
techniques 

If  it  did,  and  IF  we  could  automatically  (or  even  semi- 
automatically)  compute  multisource-based  estimates  of  these 
“states”,  we  have  not  explored  the  issue  of  how  to 
communicate  such  states  to  military  decision-makers. 


Discussion  Topics 


7-5 


Assessment  of  Need-  {or  Ontological  Development 

in  Support  of  the  Design  of  “Higher-Level” 
Information  Fusion  Processes 

Strategies  and  Technolosies/T echniaues  for 
Commimicatinn- Visualizing  Fused  Estimates  for 


“Situations”,  etc 
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General  Benefits  of  an  Ontology* 

•  Consistent  knowledge  sharing  and  reuse 

•  Improved  understandability 

•  Consensus-building 

•  Information  system  Interoperability 


•Users 

•Devel 

opers 


Basic  Ontology 


Objects/Concepts 
•Properties  and  Attributes 
•Predicate  relations 


“A  Specification  of  Concepts 
within  a  Domain” 


Statements 

And/or 

Axioms 

Applicable  in  the 
Domain 


Employed  in 
Inference  or 
Algorithmic 
Applications 
eg  L2,  L3  Fusion 


*  Eg  see  Slattery,  N.J.,  “A  Study  of  Ontology  and  its  Uses  in  Information  Technology  Systems”,  Mitre  Corp  Report 


“Non-physical  Entities  and 
the  Visualization  Challenge 


Another  “Construct”:  Operational  Readiness 

Part  of  Military  Capability: 
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-Force  Structure  (Numbers,  Composition) 

—Technical  Sophistication 

—Readiness 

—Sustainability 


Readiness  in  General 


As  Betts*  accurately  points  out,  "The  main  question  for  policy  and  strategy  should  not  be  how 
to  achieve  readiness  in  any  single  sense.  Rather,  it  is  how  to  integrate  or  balance  the  answers  to  the 
following  questions  over  a  long  period:  of  time.” 

.Readiness  for  when?  How  long  to  “ready”? 

•  Readiness  for  what?  “Ready”  to  perform  what  tasks? 

•Readiness  for  where?  “Ready”  for  what  theater  or  combat  environment? 


Time 


Task 


Environment 


Dimensions  in 
which  features  of 
Readiness  can  be 
visualized 


:  Richard  K.  Betts,  Military  Readiness:  Concepts,  Choices,  Consequences  (Washington,  D.C.:  The  Brookings  Institution,  1995),  43. 


Ontology  of  Operational  Readiness 

(a  notion) 
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Operational  Readiness 
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--  Protection  of  movement  capability  (airlanes,  sealanes,  etc) 


Many  “crisp”, 
quantitative 
elements 


On  Operational  Readiness 
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Operational 
Readiness 
Features,  to  be 
visualized  in 
“7D’  for  each 
Task  and 
Environment 
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DEFINITIONS  OF  SITUATIONAL  AWARENESS*;  MANY  VIEWS 


Definitions 

Source 

Conscious  awareness  of  actions  within  two  mutually  embedded  four-dimensionai 
envelopes. 

(Beringerand  Hancock,  1989, 

p.  646) 

A  pilot’s  continuous  perception  of  self  and  aircraft  in  relation  to  the  dynamic  envi¬ 

ronment  of  flight,  threats,  and  mission  and  the  ability  to  forecast,  then  execute 
tasks  based  on  that  perception. 

{Carroll,  1992) 

The  ability  to  extract,  integrate,  assess,  and  act  upon  task-relevant  information  is 
a  skilled  behavior  known  as  situational  awareness 

(Companion,  Corso,  Kass,  & 

Herschler,  1990) 

The  accurate  perception  of  the  factors  and  conditions  that  affect  an  aircraft  and 

its  flight  crew. 

(Edens,  1991 ,  p.  7.  Schwartz,  1993, 

uses  this  definition  with  “during  a 
defined  period  of  time”  at  the  end.) 

The  perception  of  the  elements  in  the  environment  within  a  volume  of  time  and 
space,  the  comprehension  of  their  meaning,  and  the  projection  of  their  status  in 
the  near  future. 

(Endsley,  1990,  p.  1-3) 

The  knowledge  that  results  when  attention  is  allocated  to  a  zone  of  interest  at  a 
level  of  abstraction. 

(Fracker,  1988,  p.  102) 

The  pilot’s  overall  appreciation  of  his  current ‘world.’ 

(Gibson  &  Garrett,  1990,  p.  7-1) 

One’s  ability  to  remain  aware  of  everything  that  is  happening  at  the  same  time 
and  to  integrate  that  sense  of  awareness  into  what  one  is  doing  at  the  moment. 

(Haines  and  Flateau,  1992,  p.  43) 

Where  refers  to  spatial  awareness... what  characterizes  identity  awareness,  or 
the  pilot’s  knowledge  of  the  presence  of  threats  and  their  objectives,  [as  well  as] 
engine  status  and  flight  performance  parameters.  Who  is  associated  with  respon¬ 
sibility,  or  automation  awareness  that  is  knowledge  of  who’s  in  charge.  Finally, 
when  signifies  temporal  awareness  and  addresses  knowledge  of  events  as  the 
mission  evolves. 

(Harwood,  Barnett,  and  Wickens, 

1988,  p.  316) 

The  ability  to  envision  the  current  and  near-term  disposition  of  both  friendly  and 
enemy  forces. 

(Masters,  McTaggart,  and  Green, 
1986,  p.  5;  Stiffler,  1987) 

Awareness  of  conditions  and  threats  in  the  immediate  surroundings. 

[(Monsnige  and  Retelle,  1985,  p.  92) 

*  From  Cohen,  MS,  etal.  "A  Cognitive  Framework  for  Battlefield  Commander’s  Situation  Assessment’,  U.S.  Army  Res  Inst  for  Behavior  and  Soc 
Sci.  Tech  Rpt  1 002,  July  1 994 


(From  the  Human  Engrg  literature) 


A  Notional  Processing  Chain 


Sensing 
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•bservatioi 


Multisource 

Information 

Fusion 


Presentation 

Scheme 

To 

Dec-Mkr 


Defined  components  of  the  Situation  (the  Ontology)  set  the  basis  for  : 

—Sensor  design 
—Processing  design 
—Presentation  scheme 


i.e. — the  entire  system  design 
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Another  Example:  SA  via  Plan  Recognition  Approach 

Using  an  Ontology 
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Figure  1:  An  overview  of  the  PIANETT  ontology.  Arrows  pointing  into  space  represent  relations  whose  ranges  are  not  fixed  in 
the  ontology. 


From:  PLANET:  A  Shareable  and  Reusable  Ontology  for  Representing  Plans,  Yolanda  Gil  and  Jim  Blythe 
University  of  Southern  California  /  Information  Sciences  Institute,  Marina  del  Rey,  CA,  USA 
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Consideration  of  “Definitions”  Includes  (Is?) 

the  User’s  Mental  Model 


From:  Ontology-based  Information  Visualization,  van  Harmelen  et  al,  Vrije  Universiteit,  Amsterdam 
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One  Approach:  Use  Essential 
Elements  of  Information  (EEI’s) 


Expanding  the  EEI’s 
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Another  Factor:  Temporal  Dynamics — 
Requirements  for  Visualizing  Temporal  Scenes 

User-selectable  time  gradations 

User-selectable  time  range 

User  ability  to  annotate  time  grid 

Relate  security  events  and  their  characteristics  to  time 

Relate  attack  sources  and  their  characteristics  to  time 

Relate  targeted  assets  and  their  characteristics  to  time 

Simultaneously  relate  events,  attack  sources  and  target 
characteristics  to  time 
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OK — now  assume  Level  2,  3 
Info  Fusion  Capabilities  exist 
and  have  been  developed  from 
an  Ontologically-Based 
Approach.... 

How  do  we  Visualize  these 
Complex  (“ND”)  Notions?? 


One  Approach: 
The  “Event  Wall” 

(Secure  Decisions,  Inc.) 
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5  WCenterporl 


Rear  Plane  Can  Show  Attacker 
Characteristics  or  Sensor  Sources 
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Top  View  Allows  Simultaneous  Viewing 
of  Activities  Related  to  Time 


Another  Approach: 
“Starlight” 

(Pacific  Northwest  National  Labs.) 


Exploring  and  analyzing  of  large  and  complex 
collections  of  multimedia  information,  structured  an 
unstructured  text,  geographic  information  and  digital 
imagery. 
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The  Starlight  System  Example 


From:  The  STARLIGHT  Information  Visualization  System,  JS  Risch,  DB  Rex,  ST  Dowson,  TB 
Walters,  RA  May,  BD  Moon,  Pacific  Northwest  National  Laboratory,  Richland,  Washington  USA 


|fc»TVl 


..Frequency: 


[tfiiguency:.  00006. 80MHZ 


frequency:,  37. 6MHZ 


FnJqiierfy:,  00009. 80MHZ 


fre  quency :.  4  2 . 8  M  HZ 


7-26 


The  Starlight  System  Example,  cont’d 


'.fKstjuency:, 


Figure  2.  Close-up  view  of  a  Data  Sphere  showing  structured  data  set  elements  grouped  according  to  the  values  in 
a  given  field. 
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The  Starlight  System  Example,  cont’d 
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!  B  ®  Activity  Time:010800ZSEP96 
i  B  ®  Activity  Time:010930ZSEP96 
i  B®  Activity  Time:011030ZSEP96 
|  B  9  Activity  Time:01 1 430ZSEP96 
i  B  ®  Activity  Time:012030ZSEP96 
I  B®  Activity  Time:01 21 30ZSEP96 
|  B  ®  Activity  Time:012300ZSEP96 
|  B  ®  Activity  Time:012330ZSEP96 


DRAGO  CallSign(S).  (1) 


CallSign(R)  (4  ) 
IC  ONE 


Activity.  Time.  (41) 


Unit.  Id.  (2) 


MOSTAR  BDE 


Message.  T^e.  (1) 


Message.  ID.  (41) 


Image.  Id.  (1) 


Figure  3.  Array  Set  and  corresponding  Link  Net  visualization  of  information  related  to  a  particular 
transmitting  call  sign  (DRAGO)  from  a  simulated  intelligence  database.  Note  the  temporal  distribution  of  the 
transmissions  to  receiving  call  sign  VARIC  ONE,  and  the  single  geographic  position  of  those  transmissions. 
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The  Starlight  System  Example,  cont’d 


Figure  4.  View  of  a  complex  Starlight  workspace  display  showing  multiple  simultaneous  visualization  features, 
including  a  free-text  similarity  display,  data  element  shape  and  color  encodings,  text  labels,  linkage  Tie-Nodes, 
and  linked  ancillary  information  displays. 


Class: 

A  number  of  objects  that  share  general  characteristics. 

(i.e. ,  maneuver  unit,  vehicles,  surface  ships,  fixed-wing  aircraft, 
building) 

Type: 

A  class  of  objects  that  share  particular  characteristics. 

(i.e.  tank  unit,  AAV,  destroyer,  fighter,  aircraft  hangar) 

Model: 

A  style  or  design  of  a  particular  type  of  object;  mode  of  structure 
or  formation. 

(i.e.  tank  company,  AAV  C-7,  DDG-21,  F-14,  aircraft  3rd 
echelon  maintenance  facility) 

Composition: 

The  specific  parts  or  elements  that  form  an  object.  Composition 
can  refer  to  the  material  and  structural  composition  of  a  building 
or  the  composition  of  networks. 

(i.e.  hierarchy  of  organizations,  personnel  &  equipment,  material 
structure  of  a  building) 
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Ontological  Aspects  of  Fusion 

L2  Terms 


[Definition:  Capability  is  the  capacity  and  ability  of  an  object  to  execute 
courses  of  action  or  functions  based  upon  knowledge  of  an  object’s  factors 
and  circumstances. 


Additional  Definitions: 

Ability:  Competence  and  power  to  perform  a  course  of  action. 

Capacity:  Actual  or  potential  ability  to  perform  a  course  of  action. 

Function:  The  appropriate  or  assigned  duties,  responsibilities,  missions,  or  tasks  of 

an  individual,  office,  or  organization  (facility).  (JP1-02) 

Influence:  The  capacity  or  power  of  objects  to  be  a  compelling  force  on  or  produce 
effects  on  the  actions  of  others. 

*Factors  &  circumstances  include  but  are  not  limited  to  the  following:  composition,  doctrine, 
environment,  equipment/materials,  maneuverability,  mobility,  morale,  numbers, 
readiness,  technical  sophistication,  size,  sustainability,  and  weapons  systems.  (JP1-02) 
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Ontological  Aspects  of  Fusion  L2 

Terms 


Definition:  Activity  is  the  collection  of  actions  or  events  associated  wth  an 
object.  State  is  the  status  of  an  object. 


Addtional  Definitions: 

B/ent:  A  happening  or  an  oocunenoe 

State:  The  oordtion  of  an  object  vuth  respect  to  certain  attributes  and 

arcumstanoes 

^tributes:  Qualities  or  characteristics  of  an  object 

arcumstanoes:  Condition,  detail,  or  attributes  vvith  respect  to  time,  place,  or  manner  that 

determine  an  event  or  action 

Linkage:  The  relationship  of  two  or  more  activities  which  can  be  associated  with 

an  object 

Determine:  To  settle  a  decide  a  question  by  an  authoritative  or  conclusive  decision 


